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DUAL MODE SIGNAL PROCESSING and detail than the video images. In a preferred embodiment, 

SYSTEM FOR VIDEO AND STILL IMAGE the system yields video or still data depending on the 

DATA parameters loaded into look-up-tables (LUTfe) that are used 

to configure logic circuitry for performing the image deco- 

Thts application is a continuation of Ser. No. 08/963,097 5 rrelation and entropy encoding, 
filed Nov. 3, 1997. U.S. Pat. No. 6,151,069. 

This invention is generally related to electronic imaging BRIEF DESCRIPTION OF THE DRAWINGS 

and more particularly to digital cameras. Tliese and other feauires as well as advantages of the 

The digital camera has rccenUy been developed as a ^^^^^^^ embodiments of the invention will be apparent by 

portable system that acquires and stores digital still images 10 ^ ^^^^^^^ description ^nd claims 

m electromc form. The unages may be used in a number of below where* 

different ways, such as being displayed in an "electronic** , .i^itj. 

photo album or used to embeUish graphical computer apph- FIG. 1 is a logical block diagram of a digital image 

cations. The digital camera has a user interface much like a ^.*P^ apparatus accordmg to an embodiment of the inven- 
conventional chemical film camera, but the images are i5 

captured and stored entirely using electronic solid state FIG. 2 illustrates an architecture for a signal and image 

circuitry and image processing techniques. processing system according to another embodiment of the 

A typical digital camera has an electronic image sensor invention, 

that receives incident light reflected fi-om an object or scene FIG. 3 is a logical flow diagram of signal processing steps 

through an optical interface. The optical interface may 20 performed according to another embodiment of the inven- 

include a lens system, an aperture mechanism, and perhaps tion. 
a light filter. The sensor can typicaUy be implemented as an 

array of charge coupled device (CCD) or complimentary DETAILED DESCRIPTION 

metal oxide semiconductor (CMOS) pholodetecting circuits a^u - a -lu *u- *• - j- 

. . . J > 1 . . . As briefly summarized above, the invention IS directed at 

that create light-generated signals in response to the madent 25 ^fk^j^f • 1 - „ j *, *u ♦ 

J. a method of signal processmg m a system and apparatus that 

ught. Analog signals firom the sensor are converted mto •^♦i;^ j j 

. ^ , J- • 1 / ^ /.^x J operates in at least two modes to provide still and video 

digital format by an analog-to-digital (A/D) converter and •„ .u u .u • 1 • » l 

^ c ,u 1 • . ' J, images through the same signal processing system, thereby 

arc then further processed by logic c^roiitry and/or a pm- ^ ^^^^ J^^^^^ pufcl^ing separate 

grammed process™ to yield a captured digital unage ot toe j^^^ 

cameras. In one embodiment, the 

object or scene TTie image may then be stored m loca^ 30 sig„a, processing system provides video and still data from 

memory aboard the camera. In addition, the image may be j * -i j • • i • i i_ _c 

, ' J, , .1. 1- . J. \u I^-fi the same detailed, ongmal image sensor signals by pcrft)rm- 

transfened to a computer that is Imked to the digital camera, • „ ^i^-t^i ;^orr^ ™,«h*««o iZl ^k«;„ 

c ^ 1 . - £1 J, 1- i- . • , iDg digital image processing operations m a cham rasnion. 

for storage as an electromc file and/or for further graphical ^^^^j ^^^^^ ^ 

and miage process to miprove the unage quality or use ^ ^e spaUally scaled, and then decorrelated and enaded 

wi grap cssottware. into compressed data. The different image processing opera- 

Most purchaser of digital cameras have ac«ss to a implemented in reprogrammable logic cirodtry 

desktop computer for viewing the sull images. Therefore, ^^^j,,^ ^ ^ computer bus, or alternatively by 

such purchasers migjt also enjoy using their digital camera ^ 3 ^ performance data processor to execute 

to coinmunicate with another person, such as m vidcocon- operatio^ in «)f tware. 
lerencmg with a desktop computer m order to both sec and 40 

hear the other person. However, most digital cameras arc For purposes of explanation, specific embodiments are set 

typically configured to provide only still images. Similarly, ^^'^^ P^°^^^^ ^ thorough understanding of the 

cameras used for videoconferencing do not operate as a still ">vention. However, as understood by one skilled in the art, 

camera when disconnected from the computer. Therefore, ^^"^ ^^^°g disclosure, the invention may be practiced 

there is a need for a digital camera and associated image 4S without such details. Furthermore, well-known elements, 

processing architecture that permits "dual mode" use of the P'°^ ^^^^^ i° ^^^^^ 

camera for both video, where the camera may be linked to to avoid obscurmg the invenUon. 

a desktop computer for video purposes, and still image P'^- 1 is a logical block diagram of a digital image 

capture. capture apparatus 100 such as a digital camera having a 

50 video and still processing block HO according to an embodi- 

SUMMARY pjent invention. The apparatus 100 includes an optical 

The invention is directed at a method of configuring a intcrfiacc having a lens system 104 and an aperture 108 for 

signal processing system to operate in one of at least two being exposed to incident light reflected firom an object 102 

modes to yield video and still image data, a first mode for whose image is to be captured. The apparatus 100 may also 

processing image sensor signals according to a first seleaioo ss include a strobe or electronic flash for generating supple- 

of signal processing methodologies, to obtain still image mental light to further illuminate the object 102 when the 

data, and a second mode for processing the same image apparatus 100 is operating imder low li^t conditions, 

sensor signals according to a second selection of signal The lens system 104 preferably has a fixed focal length 

processing methodologies, to obtain video image data, which may be acceptable for both video and still operation. 

In a particular embodiment of the invention, the signal 60 because a drop in the modulation transfer function ^TF) of 

processing methodologies include image scaling, the optical interface (and thus a degradation in image 

decorrelation, and entropy encoding that are performed quality) for objects at close ranges (such as a user's face 

sequentially to yield video or still data from the same during videoconferencing) may be tolerated in video mode, 

original image data. The first selection of scaling. The optical interface has an aperture mechanism 108 to 

decorrelation, and encoding is designed to provide video 65 control the dose of light on the sensor and the depth of focus, 

data, whereas the second selection is designed to provide and may be configured for both video and stiU operation 

data for stiU images which would normally have greater size with only two settings as described below. 
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The optical interface channels the incident light onto ao 
electronic image sensor 114. The image sensor 114 has a 
number of pixels which are electrically responsive to inci- 
dent light intensity and color. The sensor 114 generates 
signals that will represent a captured image having sufiBcient 
resolution to be acceptable as a still image. An A/D con- 
verter (not shown) that receives light-generated analog sen- 
sor signals may be included in the sensor 114 and used to 
generate digital sensor signals that de&ne a digital image of 
the exposed object 102 and the accompanying scene. 
Alternatively, the sensor 114 provides analog signals to the 
block 110 which in turn may perform analog signal pro- 
cessing on the signals before converting them to digital 
format. In either scenario, the digitized sensor signals thus 
define original image data which are then processed accord- 
ing to image processing methodologies by the video and still 
block 110 in order to form still images, or a sequence of 
video images depicting motion by the exposed object or 
scene, depending on whether the still or video mode of 
operation has been selected. 

Mode selection can be made by the user of the apparatus 
100 via mechanical controls (not shown) on the apparatus 
100. Mechanical knob settings can be received and trans- 
lated by a local user interface 158 into control signals and 
data to be processed by a system controller 160. 
Alternatively, the apparatus 100 can be tethered to a host 
computer, such as a Personal Computer (PC), via a host/PC 
communication interface 154, The user can then make the 
mode selection through software running on the host which 
in mm communicates the proper control signals and data to 
the system controller 160 via the host/PC interface 154. 

The system controUer 160 orchestrates the capture of 
video and still images in response to the mode selection 
made by the user as outlined above. The system controller 
160 configures the video and still processing block 110 to 
provide either still image data, or video image data depicting 
a sequence of video image frames. The images arc then 
stored aboard the apparatus 100 and/or transferred to the 
host/PC for decompression (if the images had been 
compressed), rendering, and/or display. 

The image capture apparatus 100 includes a local storage 
device 122 which receives and stores the still image data. 
The storage device 122 may include a FLASH semiconduc- 
tor memory and/or a rotating media device. The ElASH 
memory may be removable, such as the Intel® Miniature 
Card. The rotating media may also be removable or fixed, 
and may be of the magnetic disk or other type suitable for 
storing image data files. 

The image data may also be transferred outside the 
apparatus 100 via the host/PC communication interface 154. 
The communication interface 154 can be configured to 
transfer both still and video image data to the bost/PC 
according to a computer peripheral bus standard. The bus 
standards used can be, for example, the RS-232 serial 
interface, the Universal Serial Bus (USB), or the higher 
performance Institute of Electrical and Electronics Engi- 
neers (IEEE) Standard 1394-1995. 

As mentioned above, the apparatus 100 is configurable 
both optically and electronically to operate in a number of 
modes, including video mode such as during 
videoconferencing, and still mode such as when taking 
pictures similar to using a conventional portable camera. 
From an optical standpoint, a fixed focal length lens system 
104, such as one having 10 mm effective focal length, is 
preferred and can be used for both modes in order to reduce 
the cost of manufacturing the apparatus embodiment. In the 
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video mode, used in videoconferencing and rapid frame rate 
applications to capture motion, an aperture 108 of about ill 
may be selected. The prime focus for this aperture setting is 
preferably at a distance of about 1 meter for the object 102 
5 with depth of field extending to 2 meters for the background. 

In the still mode of operation, acceptable quality still 
images of both outdoor and indoor scenes can be captured. 
For indoor scenes, the light level may be so low as to require 
supplemental light generated by a strobe or electronic flash 
jg 112. One would normally select an aperture 108 of between 
ifl and for indoor scenes. Within this aperture range, the 
prime focus is at a distance of about 2 meters for the siibject 
with depth of field extending to 4 meters for the backgroimd. 
For outdoor scenes with ambient light, the prime focus is 
preferably at a distance of about 2-3 meters for the subject 
with depth of field extending to infinity. Normally, one 
would select an aperture of about £^8 for outdoor scenes to 
accomplish that focus. 

The image capture apparatus 100 can also be electroni- 
cally configured for dual mode operation by so configuring 
the video and still processing block 110 to provide either still 
image data or a sequence of video images. In one 
embodiment, the block 110 implements digital signal and 
image processing functions as logic circuitry and/or a pro- 
2^ grammed data processor to generate compressed image data 
having a predefined resolution and compression ratio from 
detailed, original image data received from the sensor 114. 
Such a block 110 is illustrated in FIG. 2 which is a logical 
block diagram of part of a digital camera (or other image 
capture apparatus) image processing system 200 according 
to another embodiment of the invention. 

FIG. 2 shows a data flow diagram for an embodiment of 
the invention of the path taken by image data for both video 
and still modes of operation. The processing block 110 
35 includes a chain of imaging function which may begin with 
a correction block 210. The correction block 210 is used 
whenever the quality of the original image data received 
from the sensor 114 warrants some sort of pre-processing 
before the image is scaled and compressed. In certain cases, 
4Q the correction block 210 performs pixel substitution, 
companding, and ganuna correction on the original image 
data received from the image sensor. The original image data 
should be of sufficient detail (e.g., 768x576 spatial resolu- 
tion or higher is preferred) to yield still images of acceptable 

45 q^^^y- 

Pixel substitution may be performed in block 210 to 
replace invalid pixel data with valid data to provide a more 
deterministic input to subsequent imaging functions. Com- 
panding may be performed to lower the resolution of each 

50 pixel (the number of bits per pbcel). For example, the 
original image data can arrive as 10 bits per pixel, whereas 
a preferred pixel resolution for the logic circuitry may be 8 
bits (1 byte). Conventional gamma conrectioo may also be 
performed to conform the information content of the image 

55 to that expected by the host computer where the image will 
be ultimately displayed. 

Other functions that may be performed in block 210 on 
each received original image frame include fixed pattern 
noise reduction which is often needed before compressing 

60 an image. Once again, whether or not any correction func- 
tions are performed by block 210 in general depends on the 
quality of the original image data received from the sensor 
114 and any subsequent image processing such as scaling or 
compression to be performed before the image data is ready 

65 for storage or transmission to the host computer. 

Once the original image data has been corrected or 
otherwise processed into the desired size or format by 
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correction block 210, the corrected data may be scaled and 
compressed if Deeded to meet the transmission and storage 
requirements of the host/PC communication interface 154 
and the local storage device 122 as seen in FIG. 1. To meet 
such requirements, the processing block 110 can include 5 
scaling and compression logic 212 to perform any necessary 
image scaling and compression prior to transmission and/or 
storage. 

For instance, the scaling and compression logic 212 may 
be configured to reduce image size and resolution to yield 
smaller, less detailed video images, as compared to larger 
and more detailed still images. Smaller and less detailed 
image data may be required in order to transmit a rapid 
sequence of video images that are to be decompressed and 
viewed in the host/PC. However, if the transmission link 
between the apparatus 100 and the host/PC has sufiScient 15 
bandwidth to transmit a sequence of detailed original image 
data at the needed rate to the host/PC, then tbe scaling and 
compression logic 212 can be simplified or even eliminated 
for both still or video operation. 

A number of digital image processing functions are con- 20 
templated for the compression logic 212 as shown in FIG. 2. 
ITiese or others similar in function may be configured as 
described below by one skilled in the art depending on the 
performance (speed of rendering the compressed image 
data) and image quality desired from the system 200 given 25 
the optical interface used in the apparatus 100. The imaging 
functions have been implemented in one embodiment as 
separate imits of logic circuitry as shown in FIG. 2 and 
which are described as follows. 

The scaling logic 214 performs a 2-D spatial scaling of the 
corrected image data in order to yield smaller images that 
may be easier to store or transmit. The scaling is done 
according to a selected scaling ratio using conventional 
known techniques. The scaling ratio may be integer or 
fractional. The scaling can be performed in a 2-dimensional 
fashion by, for instance, utilizing two separate 
1-dimcnsional scaling processes. 

The scaling logic 214 can be used for both video and still 
image capture simply by selecting the appropriate scaling 
ratio. For instance, a 4:1 sub-sampling of the corrected 
image may be performed in video mode so that 16 pixels ^ 
from the corrected image data arc averaged together to 
produce 1 pixel in the scaled image data. Based on standard 
sampUng theory, and assuming uncorrected noise sources, 
the sub-sampling may also improve the signal to noise ratio 
by or a factor of 4. Lower scaling ratios such as 2: 1 may 45 
also be used, where 4 pixels arc averaged to generate a single 
pixel in tbe scaled image data, resulting in a signal to noise 
ratio (SNR) improvement of 2. By scaling the more detailed 
corrected image data in this way during operation in video 
mode, the system 200 compensates for the increased noise 5Q 
due to lower light levels that are typically encountered with 
video operation, such as during videoconferencing. 

Next in the chain of imaging function blocks in FIG. 2 is 
the decorrelatioD and encoding logic 222. The scaled image 
data received from the scaling logic 214 is dccorrclatcd in 
preparation for entropy encoding which is a type of image 
compression, according to a selected one of a number of 
decorrelation methodologies. Once again, the user may 
select a particular decorrelation methodology that is stiitable 
for obtaining the normally smaller size video images. 

The decorrelation function can generate error image data 
as differences between neighboring pixels. One particular 
method that can be used for image decorrelation is digital 
pulse code modulation (DPCM). To obtain more compres- 
sion of the image data, if needed, for example, in transmit- 
ting a large number of video image frames, "loss" may be 65 
introduced in the form of "quantization** (mapping a first set 
of data to a smaller set of values) errors using DPCM. 



The next stape in the chain of imaging funaion blocks is 
entropy encodmg performed by block 222, which uses a 
variable length encoding technique to comprcss the decor- 
related image data. For instance, a commonly known 
entropy encoding methodology that may be used is Huffman 
encoding. Entropy encoding involves replacing symbols in 
the decorrelated image data by bit strings in such a way that 
different symbols are represented by binary strings of dif- 
ferent variable lengths, with the most commonly occurring 
symbols being represented by the shortest binary strings. 
The entropy encoding logic 222 thus provides comprcsscd 
image data, for instance as seen in FIG. 2, where tbe scaled 
8-bit data is encoded into data having variable size of 3-16 
bits. 

Once again, the encoding methodologies for obtaining 
video and still images can be different and may be selected 
depending on the mode of operation. For instance, a larger 
set of symbols (having variable binary string lengths) may 
be used for encoding still image data as compared to video 
image data. This is because therc may be more time allo- 
cated in the host/PC to decompress still images than to 
decompress video images. In contrast, for encoding video 
images, a more limited set of symbols having uniform binary 
string lengths should be employed to obtain faster decom- 
pression of a series of video image frames. In addition, 
having a uniform binary string length allows usage of a fixed 
amount of bandwidth to transmit the image data that is 
specifically suitable for a host/PC interface such as the USB. 

The image processing system 200 includes additional 
logic that facilitates the dual mode operation described 
above. In particular, the logic circuitry in blocks 210 and 212 
use programmable look-up tables (LUTs) and random access 
memories (RAMs) for flexibility in performing their respec- 
tive imaging functions. Each LUT or RAM provides infor- 
mation to its respective imaging function logic as specified 
by the selected methodology for the particular mode of 
operation. For instance, the scaling logic 214 iises a RAM 
storage arca to store intermediate scaling computations. 
Also, the LUT 234 for the decorrelation and encoding logic 
can be loaded with different rules and data required for 
performing decorrelation and encoding as known in the art, 
depending on whether a still or a video image is desired. In 
a particular embodiment, two look-up tables (LUTs) are 
used for LUT 234, one for listing the characters (a so-called 
"code book*') and one for listing the string lengths. 

Different techniques may be used to determine the proper 
values to be loaded into the RAM and LUTs. For instance, 
image metering may be performed by the camera controller 
unit 160 to determine lighting and other factors which 
impact decorrelation and entropy encoding. Also, as men- 
tioned earlier, transmission and storage constraints may 
dictate greater compression, especially during video opera- 
tion where a large number of image frames are generated, so 
that tbe LUTs for decorrelation and entropy encoding will 
include a smaller code book for compression of the image 
data. 

Although the different LUTs and RAM described above 
may be implemented as part of a single, physical RAM unit 
or alternatively may be combined in different combinations 
as one or more RAM units, each LUT and RAM is prefer- 
ably implemented in a physically separate unit to obtain 
faster performance of the imaging functions. 

After the image data has been compressed according to 
the desired mode by the compression logic 212, the now 
variable size data is then passed to the data packing unit 226 
where the data is packed into constant size, and therefore 
more manageable, data segments for more efficient storage 
and transmission over a computer bus. Once again, if the 
image data from the sensor 114 is sufficiently acceptable as 
is, and there are no further transmission or storage con- 
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straints on such data, then the data packing unit becooies 
superfluous, since the sensor image data has a constant size 
and can be easily stored or transferred outside the apparatus 
100 with minimal processing. 

Id the data packing unit 226, received data blocks of 
different size arc packed into blocks having a predefined, 
constant size. For example, the system 200 in FIG. 2, the 
data packing unit packs the variable size compressed image 
data into 16-bit blocks. The 16-bit blocks are then forwarded 
to a data flow controller 238 such as a Direct Memory 
Access (DMA) controller, which then adds address infor- 
mation to each data block before accessing a bus 242 in 
order to forward the 16-bit blocks onto the bus. The memory 
controller 246 accepts the 16-bit blocks over the bus 242 and 
stores them temporarily in memory such as dynamic RAM 
(DRAM) (not shown) aboard the apparatus 100. 

After toeing packed, the still image data may then be 
transferred over the bus 242 to the local storage 122 (see 
FIG. 1) via a local storage interface 250 coupled to the bus 
242. For instance, the local storage device 122 may be a 
removable FLASH memory card which receives the image 
data prepared as a "file", including compression tables, file 
headers, time and date stamps, and metering information 
attached to the image data. The card may then be removed 
from the apparatus 100 and inserted into a PC to transfer the 
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logic 212 in the video and still processing block HO embodi- 
ment in FIG. 2. The sequence of steps in FIG. 3 may be 
performed by the system controller 160, or by a separate, 
dedicated processor (not shown) that is also coupled to the 
bus 242. 

To summarize, the above described embodiments of the 
invention may be used in an apparatus such as a portable 
digital camera that operates in both still mode (for capturing 
still images as a portable camera) and in video mode (where 
the digital camera is tethered via a computer peripheral bus 
interface to a host computer or other image viewing system). 
The camera has an image sensor and a video and still 
processing blodc HO that is configured to capture detailed 
images in still mode, in both outdoor and indoor scenes. In 
video mode, the camera may be configured to compress a 
sequence of the detailed images (if needed for transmission 
and storage) using the same processing block 110 in order to 
capture a video sequence that can be transmitted to the host 
computer for viewing via a computer peripheral bus inter- 
face. 

The embodiments of the invention described above are, of 
course, subject to some variations in structure and imple- 
mentation. For instance, although the image data path in the 
processing block HO is shown as being 8 bits wide initially 
and up to 16 bits when compressed, one skilled in tl^ art will 



still image data for decompression, viewing and/or further 25 recognize that the invention can be implemented using other 
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processing in the PC. 

As an alternative to using a removable storage device, the 
host/PC communication controller 154 may be used to 
transfer both the still and video images outside the apparatus 
100. This may be accomplished by preparing the still image 30 
data as a disk file appropriate for transmission using the 
particular bus standard used in the communication interface 
154, to be transferred to the host computer for storage and 
access by a host processor (not shown). The video image 
data can be streamed according to known techniques to the 35 
host computer via a controller interface such as USB. 

The dual mode operation of the image capture apparatus 
100 and processing system 200 has been described in the 
context of a bus-based architecture shown in FIG. 2. To 
further facilitate software control of the different modes of 40 
operation in this architecture, a number of memory-mapped 
control registers (not shown) may be coupled to the bus 242 
to allow the system controller 160 to configure the apparatus 
100 and the system 200 with the desired mode of operation. 
Instructions can be provided for execution by the system 
controller to access the LUTs, RAM, and control registers 
via the bus 242 in order to program the parameters needed 
for the proper image processing methodobgies of the 
selected mode of operation. For instance, the different rules 
and parameters for scaling, deoorrelation, and entropy 
encoding methodologies for all modes of operation may be 
stored as controller instructions aboard the apparatus 100 
during manufacture, where each mode of operation is 
assigned a different set of methodologies. The appropriate 
set can be loaded into the video and still block 110 in 
response to a mode selection by the user, made through 55 
either the local user interface 158 or the host/PC commu- 
nication interface 154. 

Although the current preferred embodiment of the video 
and still block 110 is as logic circuitry, the image processing 
system 200 may also be equipped with a programmed high 60 
performance processor executing instructions to perform the 
digital imaging functions of the block 110. Exemplary steps 
that may be performed by such a processor are illustrated in 
FIG. 3, and can be easily understood based on the descrip- 
tion above of the correction block 210 and the compression 



data path widths. Also, the system controller 160 may be 
combined with the data flow controller 238 into one 
physical, integrated circuit unit such as a microcontroller. 
Therefore, the scope of the invention should be determined 
not by the embodiments illustrated but by the appended 
claims and their legal equivalents. 
What is claimed is: 

1. An image processing system comprising: 

signal processing circuitry that is configurable to operate 
in one of at least two modes to yield video and still 
image data, the processing circuitry including scaling 
logic for spatially scaling image data related to sensor 
signals into scaled image data, and 

compression logic for compressing the scaled image data 
into compressed image data, 

wherein in the first mode, the scaling logic is config- 
ured to scale according to a first scaling methodology 
suitable for generating video image data, and the 
compression logic is configured to compress accord- 
ing to a first compression methodology suitable for 
generating video image data, and 

wherein in the second mode, the scaling logic is con- 
figured to scale according to a second scaling meth- 
odology suitable for generating still image data, and 
the compression logic is configured to compress the 
scaled image data according to a second compression 
methodology suitable for generating still image data. 

2. The image processing system of claim 1 further com- 
prising; 

a compression look-up table for providing information to 
the compression logic, as specified by the first and 
second methodologies. 

3. The image processing system of claim 2 further com- 
prising: 

a data packing unit for packing the compressed data into 
packed image data having a predefined and constant 
size. 
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